Skip to content

feat(benchmark): Crossover concurrency: thread pool + rate-limit guard + thread-safe sidecar (byte-identical)#23

Merged
tcballard merged 2 commits into
mainfrom
claude/decisiongrounding-concurrency
Jul 9, 2026
Merged

feat(benchmark): Crossover concurrency: thread pool + rate-limit guard + thread-safe sidecar (byte-identical)#23
tcballard merged 2 commits into
mainfrom
claude/decisiongrounding-concurrency

Conversation

@tcballard

@tcballard tcballard commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

A full real crossover runs ~15h sequentially; the answering/embedding calls are I/O-bound, so running N cells in flight cuts wall-clock to roughly 1/N. Delivered in the three pieces requested, and proven byte-identical to sequential on a small grid. Full suite green (477 passed, 6 skipped).

1. Concurrency pool (thread pool, not async)

build_dataset is refactored into two phases:

  • Phase 1 — compute each cell's outcome, optionally on a ThreadPoolExecutor(max_workers=concurrency). Each worker uses a thread-local answering model so the mutable last_usage never races across cells. Cell computation is a pure function of (seed, N, arm, scenario) — the corpus is deterministic — so outcomes don't depend on completion order.
  • Phase 2 — assemble the dataset in deterministic grid order (the exact old sequential accumulation).

Because assembly is order-deterministic, the dataset is byte-identical for any concurrency. Threaded through run_seeds / build_dataset_multiseed; default concurrency=1 leaves behavior unchanged. (The batched builder already parallelises server-side via the Batch API, so it's untouched.)

2. Rate-limit guard

util/ratelimit.py — a thread-safe token-bucket RateLimiter(rpm) acquired once per cell, so the pool self-throttles under a hosted API's requests-per-minute cap. It only delays calls, never changes what one returns, so results stay byte-identical.

3. Thread-safe sidecar writes

The durable .partial.jsonl sidecar is now written from worker threads, so _on_cell serialises its whole write+flush+print under a lock — interleaved writes would corrupt the JSONL that --resume reads back.

CLI

demo gains --concurrency N (cells in flight) and --rpm RATE (token-bucket cap), threaded into the single-seed, multi-seed, and --augment paths.

Proof (the deliverable)

tests/test_concurrency.py:

  • byte-identical on a 2×3 grid (2 N × 3 discriminating scenarios × 2 arms): concurrency=1 vs concurrency=6 → identical json.dumps(sort_keys=True); same for multi-seed (1 vs 8).
  • CLI end-to-end: a --concurrency 6 demo run produces a byte-identical crossover_dataset.json and an uncorrupted sidecar (every line valid JSON, every cell exactly once — no interleaving).
  • rate limiter throttles (drained bucket → 4 acquisitions at 2/s take ≥1.5s) and rejects non-positive rpm.
  • The existing crossover/stats/resume tests still pass, confirming the two-phase refactor is behavior-preserving.

Independent of PR #20 (roster Step 1) and PR #22 (Holm/CI-clamp); branched from main.

tcballard added 2 commits July 9, 2026 10:16
…ntical)

A full real crossover runs ~15h sequentially; the answering/embedding
calls are I/O-bound, so running N cells in flight cuts wall-clock to a
fraction. build_dataset is refactored into two phases: Phase 1 computes
each cell's outcome (optionally on a ThreadPoolExecutor, each worker using
a thread-local answering model so the mutable last_usage never races);
Phase 2 assembles the dataset in deterministic grid order. The result is
BYTE-IDENTICAL to sequential for any concurrency -- proven on a 2x3 grid
(concurrency 1 vs 6) and multi-seed (1 vs 8).

util/ratelimit.py adds a thread-safe token-bucket RateLimiter acquired
once per cell so the pool self-throttles under a hosted API's RPM cap; it
only delays, never changes a call's result. concurrency/rate_limiter
thread through build_dataset / run_seeds / build_dataset_multiseed
(default concurrency=1 -> unchanged behavior).
The crossover sidecar is now written from worker threads, so _on_cell
serialises its whole write+flush+print under a lock -- interleaved writes
would corrupt the JSONL that --resume reads back. The demo command gains
--concurrency N (cells in flight) and --rpm RATE (token-bucket cap),
threaded into build_dataset / build_dataset_multiseed / the augment
run_seeds path. Proven end-to-end: a --concurrency 6 CLI run produces a
byte-identical crossover_dataset.json and an uncorrupted sidecar (every
line valid JSON, every cell exactly once).
@tcballard tcballard changed the title Crossover concurrency: thread pool + rate-limit guard + thread-safe sidecar (byte-identical) feat(benchmark): Crossover concurrency: thread pool + rate-limit guard + thread-safe sidecar (byte-identical) Jul 9, 2026
@tcballard
tcballard merged commit a5e241f into main Jul 9, 2026
7 checks passed
@tcballard
tcballard deleted the claude/decisiongrounding-concurrency branch July 9, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant